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Abstract- In  this  paper,  a  “Surgeon  Assistant  System”  using 
Enhanced  Reality  (ER)  approach  is  presented.  The  main  idea  is 
to  use  Virtual  Reality  (VR)  device(s),  and  assist  the  surgeon  with 
one  and  two-dimensional  information  of  human  body  and  three- 
dimensional  volumetric  surface  models  of  anatomic  tissues, 
especially  in  the  brain  operations.  This  opportunity  may 
increase  the  concentration  of  the  surgeon  with  the 
combinational  use  of  real  and  computer  generated  information 
combination.  Therefore,  increased  concentration  and  well- 
organized  information  improve  the  success  rate  of  the  surgery. 
This  study  covers  hardware  and  software  implementation  of  a 
prototype  system  recommending  requirements  of  such  tools. 
Keywords  -  Computer  Aided  Surgery  (CAS),  Enhanced  Reality. 

I.  Introduction 

Surgery  is  a  treatment  method  for  some  abnormal 
structures  in  any  organ,  tumor  and  deformations.  These 
illnesses  can  be  replaced  or  corrected  in  the  surgeries.  All 
surgeries  have  risks  about  the  health  of  the  patient 
depending  on  the  organ  and  the  type  of  operation;  therefore 
high  concentration  and  care  are  required.  In  traditional 
procedures,  a  surgeon  has  to  prepare  thoroughly  for  the 
operation.  Although  operation  rooms  are  very  complex  areas 
because  of  the  need  for  various  high  technology  equipments, 
the  success  of  the  operation  depends  primarily  on  the 
performance  of  the  surgeon.  The  surgeons  of  today,  as  in  the 
past  rely  primarily  on  hand-eye  coordination  for  performing 
various  procedures.  The  surgeons  generally  complain  from 
restricted  field  of  view,  lack  of  contrast  between  target  and 
surrounding  tissues  and  inability  to  see  beneath  the  surface 
[1].  Computer  Assisted  Surgery  (CAS)  systems  can  solve 
these  problems  using  high-resolution  imaging  and  detailed 
anatomical  information  combination.  The  surgeon's  vision 
can  be  extended  by  the  use  of  MRI  and  CT,  two  powerful 
imaging  technologies  which  can  penetrate  beneath  the 
surface  and  provide  an  image  of  the  human  body  in  three 
dimensions  [2].  Using  computerized  reconstruction  and 
image  processing  techniques/images  can  be  provided  in  a 
form  in  which  the  surgeon,  using  an  interactive  computer 
workstation,  can  navigate  through  the  simulated  organs 
without  cutting,  and  can  plan  and  rehearse  surgical 
procedures.  Three-dimensional  recontructions  of  imaging 
data  can  also  be  used  for  real-time  control  during  the 
performance  of  surgical  procedures  or  other  therapeutical 
interventions  [3]. 

Our  approach  shown  in  Fig  1.  Is  based  on  the  Enhanced 
Reality  methodology  to  assist  the  surgeon  with  the 
combination  of  real  video  and  computer  generated  two 
dimensional  slice  views  and  three  dimensional  surface 
models  of  the  organs  in  the  education,  planning  and  intra¬ 
operative  phases  of  the  surgeries. 


Fig.  1.  Basic  Enhanced  Reality  Approach  for  CAS  Applications 
Revised  From  Reference  [4] . 


The  main  goal  in  this  study  is  to  increase  the  concentration 
and  compactness  of  the  information  of  the  surgeon  and 
increase  the  success  rate  of  the  operation.  This  may  help  to 
improve  the  human  health. 

II.  Method 

Our  Method  requires  hardware  and  software  implementation 
of  previously  stored  volumetric  data  and  realtime  acquired 
information.  Therefore,  the  system  includes  several  main 
sections  shown  in  Fig  2:  Data  Acquisition,  Data 
Manipulation,  Visualization  and  Medical  Implementation. 

1)  Data:  The  system  can  process  one  dimensional  vital 
signs  from  human  body  such  as  temperature,  blood  pressure 
acquired  from  patient  monitor,  two  dimensional  slice  views 
of  NMR  and  CT  images  (slice),  and  three  dimensional 
arranged  NMR  and  CT  volumes.  This  volumetric  information 
can  be  acquired  slice  by  slice  using  fast  equipment.  We  have 
different  slice  sized  and  different  slice  numbered  information. 
These  properties  can  be  changed  regarding  to  operation  area 
and  operation  sensitivity.  For  example,  brain  surgeries  need 
maximum  detail,  so  the  number  of  slices  must  be  the  highest. 

2)  Data  Manipulation:  In  this  study,  the  main  idea  is  to 
use  volumetric  surface  models  of  the  medical  tissues  with  the 
real  data,  because,  direct  renderings  of  volumes  do  not  give 
efficient  information  about  the  anatomy  and  require  much 
processing  resources.  Thus,  we  have  to  process  raw  data  in 
order  to  reconstruct  a  surface  model  of  the  volume  of  interest 
(VOI)  and  make  visualization  phase  easier  and  faster.  The 
processing  phase  must  have  three  steps:  Preprocessing  step 
for  data  conditioning,  Processing  step  for  NMR 
Segmentation  and  3D  surface  Reconstruction  from 
previously  processed  and  segmented  data. 
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Fig.  2.  Basic  Block  Diagram  for  Suggested  and  Implemented  System  [5]. 

In  preprocessing  step,  various  fundamental  image  processing 
algorithms  are  performed.  For  example,  slice  data  has  a 
background  and  noise  which  must  be  cancelled.  By  using  a 
thresholding  scheme,  interested  tissues  can  be  recovered 
easily.  For  a  CT  image,  thresholding  is  sufficient  to  differ 
bony  media  from  soft  tissues,  but,  for  an  NMR  slice,  his  type 
of  basic  operation  is  not  capable  of  classifying  all  brain 
tissues.  So,  an  advanced  segmentation  scheme  must  be 
applied  to  NMR  data.  Before  segmentation,  we  had  required 
a  normalization  of  data  for  standard  processing  routine.  For 
that  purpose,  we  have  applied  contrast  normalization  via 
histogram  equalization  and  tone  normalization  by  using 
converting  all  volumetric  data  to  8  bit  boundaries  from  12  or 
16  bit  data  depths. 

NMR  segmentation  plays  an  important  role  in  our  work, 
because,  classifying  brain  tissues  from  NMR  slices  requires 
an  automatic  segmentation  scheme.  Although,  some  studies 
are  announced  as  automatic  segmentation  method  [6],  [7], 
[8],  no  method  is  capable  of  fully  automatic  segmentation 
process,  because  every  method  is  limited  with  the  data 
acquisition  method,  biological  structure  and  modality.  So,  we 
had  to  develop  a  segmentation  process  for  our  system. 

Several  approaches  for  MRI  segmentation  are  seen  in  the 
literature.  These  are  traditional  spatial  methods  based  on 
region  growing  and  edge  detection  [9],  statistical  Methods 
[10]  use  statistical  properties  of  tissues,  neural  methods  [11], 
[6],  [8]  and  fuzzy  methods.  At  the  same  time,  combined 
approaches  are  developed  to  bring  together  advantages  of 
several  segmentation  methods  in  order  to  obtain  finest 
results.  Our  approach  is  developed  with  this  aim.  We  have 
developed  a  neural  method  which  uses  spatial  and  statistical 


properties  of  medical  tissues  for  performing  segmentation  of 
brain  structures. 

Our  segmentation  process  uses  Self  Organizing  Feature 
Maps  (SOFM)  [12].  In  SOM,  on  the  contrary  to  Feedback 
and  feedforward  neural  networks,  neighboring  cells  in  a 
neural  network  compete  in  their  activities  by  means  of 
mutual  lateral  interactions,  and  develop  adaptively  into 
specific  vectors  of  different  signal  patterns.  In  this  category 
learning  is  called  competitive,  unsupervised  or  self¬ 
organizing.  This  property  makes  this  method  the  most  proper 
approach  for  automatic  segmentation,  because,  supervising 
phase  goes  shorter  or  is  consisting  of  efforts  on  only  labeling 
tissues  from  trained  network. 

During  the  study  we  have  tried  different  map  sizes  and 
features  to  train  and  test  the  network.  5x5  to  10x10  map  sizes 
are  used.  The  best  results  are  obtained  by  using  7x7 
rectangular  map.  For  train  and  testing  phases,  38  different 
features,  which  are  spatial  and  statistical,  were  applied  to  our 
network.  Spatial  features  are  position,  index  tone,  index 
neighbourhood  tones,  neighbourhood  gradient,  5x5  window 
average.  Statistical  features  given  below  are  Energy, 
Momentum  and  Entropy  [13]  properties  of  selected  index 
window.  These  properties  are  related  to  texture  on  the  image. 
For  example,  entropy  is  directly  related  to  amount  of  changes 
in  our  texture.  Homogenous  images  give  low  entropy  values. 
Thus,  NMR  image,  which  is  seen  as  textured  structure,  can 
be  understood  differently. 


Energy=  £  £  M]  (/,  j)  (1) 

*  J 

Momentum=  £  £  (/  -  A  M<i  (*\  j)  (2) 

i  J' 

Entropy  =  £  £  Md  (/,  j )  In  Md  (i,  j )  (3) 

i  J' 

Where  M d(/j)  is  Cooccurance  Matrix  of  selected  window. 

After  the  preprocessing  and  processing  stages,  related 
medical  brain  tissues,  which  are  skull,  white  matter,  gray 
matter  and  pathology  (tumor),  are  segmented  and  prepared 
for  modeling. 

Modeling  phase  consists  of  isosurface  extraction  method 
based  on  finding  discontinuities  over  an  isovalued  volume 
and  determination  of  surface  patches  (triangles  or  tethedras) 
coating  closed  media  having  almost  the  same  isovalue. 

For  extraction  of  isosurfaces,  two  main  algorithms  are 
available:  Marching  Cubes  [14]  and  Delaunay  Triangulation. 
Marching  cubes  algorithm  has  a  relatively  easier 

implementation  procedure  than  Delaunay  Triangulation. 
Delaunay  triangulation  is  based  on  voronoi  diagrams  and 
requires  complex  computations.  Thus,  we  have  performed 
Marching  Cubes  Algorithm  because  of  their  easier 

implementation  and  high  speed. 
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Fig.  5.  Video  Overlay  Method 
III.  RESULTS 

During  the  study,  much  effort  has  been  spent  on  MRI 
segmentation,  three-dimensional  surface  modeling  of  brain 
tissues  and  interactive  visualization  of  models  on  real  video. 
Fig. 6  shows  examples  of  the  results  of  segmentation  phase. 
After  segmentation,  synthetic  surface  patches  for  example 
unit  cube  and  reconstructed  surface  of  human  brain  white 
matter  can  be  seen  in  Fig. 7. 

During  the  surgery,  our  system  provides  three  types  of 
information.  One  of  which  is  one-dimensional  vital  signs 
from  human  body  such  as  temperature  and  ECG  information. 
This  information  can  be  supplied  by  a  conventional  hardware 
and  it  can  be  monitored  real-time  on  the  HMD  in  order  to 
recover  any  changes  of  the  patient.  The  second  information  is 
two  dimensional  slice  views  of  the  NMR  and  CT  data.  The 
surgeon  may  need  any  cross  section  of  the  body  in  the 
operation.  Fig. 8  shows  one  and  two  dimensional  data 
monitoring  on  real  video  in  our  system. 


Marching  Cubes  Algorithm  is  a  divide  and  conquer 
algorithm,  uses  two  main  lookup  tables  and  decides  which 
surfaces  patches  are  convenient  for  selected  cube 
arrangement  as  shown  in  Fig. 3.  There  are  256  different 
arrangement  possibilities  for  any  selected  unit  cube.  By  using 
symmetry,  these  are  eliminated  to  15  cases  given  in  Fig.4. 

3)  Visualization:  Segmented  and  modeled  three 
dimensional  information  is  ready  to  be  displayed  in  a  three 
dimensional  viewport.  In  order  to  increase  the  speed  of  the 
vision  and  make  our  visualization  scheme  platform 
independent,  we  have  used  OpenGL  graphics  library.  This 
direct  interface  to  graphics  hardware  provides  an  efficient 
interactive  manipulation  of  our  models.  The  user  can 
navigate  through  the  virtual  world,  rotate,  zoom  and  cut  the 
model. 

Second  operation  in  visualization  section  is  video  overlay 
method  Combination  of  virtual  world  and  real  running  video 
is  realized  by  special  hardware,  a  frame  grabber  with  the 
function  key  overlay.  Doing  this  operation  by  hardware 
prevents  the  computation  power  lacks  during  operation. 
Thus,  all  computation  efforts  can  be  focused  on  interactive 
display  options.  Video  overlay  method  is  shown  in  Fig. 5. 


Fig. 6.  Segmentation  Result  for  White  and  Gray  Matter 


(a)  (b) 

(b) 

Fig.  7.  (a)  Generated  Surface  Patches  for  Example  Unit  Cube,  (b) 
Reconstructed  Human  Brain  White  Matter 
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The  most  important  display  content  in  this  work  is  three- 
dimensional  surface  model  overlay  procedure.  This 
opportunity  differs  our  work  from  conventional  surgery 
assistance  methods.  Usage  of  three  dimensional  views  of  the 
media  and  manipulation  interactively  gives  maximum 
accuracy.  Navigation  fly-through  and  marking  options  may 
be  helpful  for  the  success  of  the  surgery.  Three-dimensional 
model  overlay  option  used  in  the  operation  is  given  as  Fig. 9. 

IV.  Conclusion 

This  method  for  surgery  assistance  is  to  reconstruct  a 
three  dimensional  surface  models  of  the  medical  tissues  and 
display  these  renderings  and  other  one-  and  two-dimensional 
conventional  information  on  an  HMD  (Head  Mounted 
Display)  which  is  mounted  on  the  surgeon's  head.  This 
possibility  introduces  the  maximum  benefits  of  a  surgery 
assistance  system  by  using  real-time  interactive  information 
support 

In  this  study,  a  surgery  planning,  education  and  surgery 
assistance  system  has  been  developed.  The  approach  is 
Enhanced  Reality  (ER),  donated  by  an  HMD  and  a  camera, 
and  the  combination  of  real  world  and  real  data.  The  system 
was  evaluated  for  the  brain  operations,  but,  it  can  be  used  for 
any  surgery  of  any  area  of  human  body. 

Such  systems  can  be  used  for  educative  purposes  without 
any  doubt  for  student  and  the  patient.  Studying  computer 
models  and  performing  virtual  surgeries  improves  the 
experience  required  for  a  surgeon.  Also  this  type  of 
assistance  may  increase  the  concentration  of  the  surgeon 
during  the  surgery  and  may  cause  improvements  of  the 
success  rate.  Therefore,  we  may  prevent  possible  failures. 

Comments  given  for  this  system  by  the  surgeons  indicate 
that  this  system  may  be  useful  in  surgical  actions.  Some  of 
the  operators  believe  that  this  system  is  a  new  patient  monitor 
and  a  surgery  assistance  tool. 

Our  system  will  be  our  first  step  to  telemedicine.  In  the 
future,  this  system  may  be  used  to  share  surgical  data, 
including  patient  vital  signs  and  surgeon’s  treatment, 
therefore  it  could  be  possible  to  realize  the  surgeries  with  the 
help  of  remote  experts  consultance. 


Fig.  8  (a)  One  and  two  Diemnsional  Information  assistance 


Fig.9  Three-Dimensional  View  of  Compuer  Generated  Transparent  Model 
and  Video  Overlay  Operation 
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